A 3D Head Tracker for an Automatic Lipreading System

A real world automatic lip reading system must be able to cope with movement of the speaker’s head during operation. The observed mouth shape depends not only on the true shape of the mouth, but also the angle at which the mouth is viewed. As the speaker’s head moves and rotates the viewing angle changes. The resulting distortion can lead to inaccurate mouth measurement and incorrect phoneme recognition. We have developed a system that robustly measures the dimensions of a speaker’s mouth whilst the speaker’s head is moving and exhibiting rotations of up to 30 degrees away from the camera. Our system tracks the pose of the speaker’s head in 3D, detects the mouth by tracking unadorned lip contours and estimates the 3D locations of the upper and lower lip edges and the mouth corners. The system is demonstrated on a person speaking whilst moving his head in 3D, and the mouth height and width are corrected over 9 seconds of 25Hz video footage.